

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Notes and Thoughts on Cephadm’s scalability &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/underscore.js"></script>
        <script src="../../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="Ceph 存储集群" href="../../../rados/" />
    <link rel="prev" title="Storage Devices and OSDs Management Workflows" href="../design/storage_devices_and_osds/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../../../cephadm/">Cephadm</a> &raquo;</li>
        
          <li><a href="../">CEPHADM 开发者文档</a> &raquo;</li>
        
      <li>Notes and Thoughts on Cephadm’s scalability</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../../_sources/dev/cephadm/scalability-notes.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../">
          

          
            
            <img src="../../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../developer_guide/">开发者指南</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../developer_guide/intro/">简介</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../developer_guide/essentials/">必备知识</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../developer_guide/merging/">何时、合并了什么</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../developer_guide/issue-tracker/">问题追踪器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../developer_guide/basic-workflow/">基本工作流</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../developer_guide/tests-unit-tests/">测试：单元测试</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../developer_guide/testing_integration_tests/">测试：集成测试(Teuthology)</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../developer_guide/running-tests-locally/">测试：在本地运行测试</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../developer_guide/dash-devel/">Ceph Dashboard 开发者文档 (之前是 HACKING.rst)</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../developer_guide/jaegertracing/">Tracing 开发者文档</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">Cephadm 开发者文档</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../developing-cephadm/">Developing with cephadm</a></li>
<li class="toctree-l3"><a class="reference internal" href="../host-maintenance/">Host Maintenance</a></li>
<li class="toctree-l3"><a class="reference internal" href="../compliance-check/">Compliance Check</a></li>
<li class="toctree-l3"><a class="reference internal" href="../design/storage_devices_and_osds/">存储设备和 OSD 管理</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">Notes and Thoughts on Cephadm’s scalability</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#about-this-document">About this document</a></li>
<li class="toctree-l4"><a class="reference internal" href="#intro">Intro</a></li>
<li class="toctree-l4"><a class="reference internal" href="#considerations-of-cephadm-exporter-s-rest-api">Considerations of cephadm-exporter’s REST API</a></li>
<li class="toctree-l4"><a class="reference internal" href="#mon-s-config-key-store">MON’s config-key store</a></li>
<li class="toctree-l4"><a class="reference internal" href="#increase-the-worker-pool-size">Increase the worker pool size</a></li>
<li class="toctree-l4"><a class="reference internal" href="#backwards-compatibility">Backwards compatibility</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="notes-and-thoughts-on-cephadm-s-scalability">
<h1>Notes and Thoughts on Cephadm’s scalability<a class="headerlink" href="#notes-and-thoughts-on-cephadm-s-scalability" title="Permalink to this headline">¶</a></h1>
<div class="section" id="about-this-document">
<h2>About this document<a class="headerlink" href="#about-this-document" title="Permalink to this headline">¶</a></h2>
<p>This document does NOT define a specific proposal or some future work.
Instead it merely lists a few thoughts that MIGHT be relevant for future
cephadm enhacements.</p>
</div>
<div class="section" id="intro">
<h2>Intro<a class="headerlink" href="#intro" title="Permalink to this headline">¶</a></h2>
<p>Current situation:</p>
<p>Cephadm manages all registered hosts. This means that it periodically
scrapes data from each host to identify changes on the host like:</p>
<ul class="simple">
<li><p>disk added/removed</p></li>
<li><p>daemon added/removed</p></li>
<li><p>host network/firewall etc has changed</p></li>
</ul>
<p>Currently, cephadm scrapes each host (up to 10 in parallel) every 6
minutes, unless a refresh is forced manually.</p>
<p>Refreshes for disks (ceph-volume), daemons (podman/docker), etc, happen
in sequence.</p>
<p>With the cephadm exporter, we have now reduced the time to scan hosts
considerably, but the question remains:</p>
<p>Is the cephadm-exporter sufficient to solve all future scalability
issues?</p>
</div>
<div class="section" id="considerations-of-cephadm-exporter-s-rest-api">
<h2>Considerations of cephadm-exporter’s REST API<a class="headerlink" href="#considerations-of-cephadm-exporter-s-rest-api" title="Permalink to this headline">¶</a></h2>
<p>The cephadm-exporter uses HTTP to serve an endpoint to the hosts
metadata. We MIGHT encounter some issues with this approach, which need
to be mitigated at some point.</p>
<ul class="simple">
<li><p>With the cephadm-exporter we use SSH and HTTP to connect to each
host. Having two distinct transport layers feels odd, and we might
want to consider reducing it to only a single protocol.</p></li>
<li><p>The current approach of delivering <code class="docutils literal notranslate"><span class="pre">bin/cephadm</span></code> to the host doesn’t
allow the use of external dependencies. This means that we’re stuck
with the built-in HTTP server lib, which isn’t great for providing a
good developer experience. <code class="docutils literal notranslate"><span class="pre">bin/cephadm</span></code> needs to be packaged and
distributed (one way or the other) for us to make use of a better
http server library.</p></li>
</ul>
</div>
<div class="section" id="mon-s-config-key-store">
<h2>MON’s config-key store<a class="headerlink" href="#mon-s-config-key-store" title="Permalink to this headline">¶</a></h2>
<p>After the <code class="docutils literal notranslate"><span class="pre">mgr/cephadm</span></code> queried metadata from each host, cephadm stores
the data within the mon’s k-v store.</p>
<p>If each host would be allowed to write their own metadata to the store,
<code class="docutils literal notranslate"><span class="pre">mgr/cephadm</span></code> would no longer be required to gather the data.</p>
<p>Some questions arise:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">mgr/cephadm</span></code> now needs to query data from the config-key store,
instead of relying on cached data.</p></li>
<li><p>cephadm knows three different types of data: (1) Data that is
critical and needs to be stored in the config-key store. (2) Data
that can be kept in memory only. (3) Data that can be stored in
RADOS pool. How can we apply this idea to those different types of
data.</p></li>
</ul>
</div>
<div class="section" id="increase-the-worker-pool-size">
<h2>Increase the worker pool size<a class="headerlink" href="#increase-the-worker-pool-size" title="Permalink to this headline">¶</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">mgr/cephadm</span></code> is currently able to scrape 10 nodes at the same time.</p>
<p>The scrape of a individual host takes the same amount of time persists.
We’d just reduce the overall execution time.</p>
<p>At best we can reach O(hosts) + O(daemons).</p>
</div>
<div class="section" id="backwards-compatibility">
<h2>Backwards compatibility<a class="headerlink" href="#backwards-compatibility" title="Permalink to this headline">¶</a></h2>
<p>Any changes need to be backwards compatible or completely isolated from
any existing functionality. There are running cephadm clusters out there
that require an upgrade path.</p>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../../../rados/" class="btn btn-neutral float-right" title="Ceph 存储集群" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../design/storage_devices_and_osds/" class="btn btn-neutral float-left" title="Storage Devices and OSDs Management Workflows" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>